{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bee64cb2-8555-4239-babc-8f0d249bce19",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/mrdbourke/pytorch-deep-learning/blob/main/extras/pytorch_cheatsheet.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b721b6b3-bf94-4f8f-9aae-9fa119962b1d",
   "metadata": {},
   "source": [
    "# PyTorch Cheatsheet\n",
    "\n",
    "Some of the most commonly used commands/setups in PyTorch.\n",
    "\n",
    "> **Note:** One of the best ways to get help for PyTorch specific functions and use cases is to  search \"pytorch how to make a convolutional neural network\" or \"pytorch transformer layers\" or \"pytorch loss functions\". I do this regularly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ee9c36f-3fd9-42b8-a5b6-a5a4de1e5cdd",
   "metadata": {},
   "source": [
    "## Imports\n",
    "\n",
    "You can install PyTorch on various platforms via the [PyTorch installation page](https://pytorch.org/get-started/locally/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "98cf25ad-8e15-4692-a496-dc69b83a2f00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch version: 1.13.1\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# Check the version\n",
    "print(f\"PyTorch version: {torch.__version__}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9e369e44-8cea-4171-87f4-93bf421155c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'torch.nn.modules.linear.Linear'>\n"
     ]
    }
   ],
   "source": [
    "# Can also import the common abbreviation \"nn\" for \"Neural Networks\"\n",
    "from torch import nn\n",
    "\n",
    "# Almost everything in PyTorch is called a \"Module\" (you build neural networks by stacking together Modules)\n",
    "this_is_a_module = nn.Linear(in_features=1,\n",
    "                             out_features=1)\n",
    "print(type(this_is_a_module))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d89b206a-6869-4d33-b313-e952aef82adf",
   "metadata": {},
   "source": [
    "### Data imports\n",
    "\n",
    "Since most of machine learning is finding patterns in data, it's good to know how to work with datasets in PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5a6cf16f-f713-4a5a-8666-a622695eb612",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PyTorch Dataset (you can store your data here) and DataLoader (you can load your data here)\n",
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58418977-bdb1-4dfb-858c-3d99e5d0ef77",
   "metadata": {},
   "source": [
    "## Creating Tensors\n",
    "\n",
    "One of the main use cases of PyTorch is for accelerated deep learning computing.\n",
    "\n",
    "And deep learning usually involves the manipulation of large tensors (big, multi-dimensional collections of numbers).\n",
    "\n",
    "PyTorch has a number of methods to create tensors.\n",
    "\n",
    "> **Note:** For a more extensive overview of creating tensors with PyTorch, see [00. PyTorch Fundamentals](https://www.learnpytorch.io/00_pytorch_fundamentals/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ce6c9952-5783-4ade-8c92-06b4393139ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a single number tensor (scalar)\n",
    "scalar = torch.tensor(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f3f4f286-a8e6-4db0-8d5e-0c7aaf3b7c2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a random tensor\n",
    "random_tensor = torch.rand(size=(3, 4)) # this will create a tensor of size 3x4 but you can manipulate the shape however you want"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bf17bde1-c03d-4f7c-ab2a-d24f59161cf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multiply two random tensors\n",
    "random_tensor_1 = torch.rand(size=(3, 4))\n",
    "random_tensor_2 = torch.rand(size=(3, 4))\n",
    "random_tensor_3 = random_tensor_1 * random_tensor_2 # PyTorch has support for most math operators in Python (+, *, -, /)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f51efe1-609a-40c0-94a2-1f6c816599ef",
   "metadata": {},
   "source": [
    "## Domain Libraries\n",
    "\n",
    "Depending on the specific problem you're working on, PyTorch has several domain libraries.\n",
    "\n",
    "- **[TorchVision](https://pytorch.org/vision/stable/index.html)** — PyTorch’s resident computer vision library. \n",
    "- **[TorchText](https://pytorch.org/text/stable/index.html)** — PyTorch’s in-built domain library for text.\n",
    "- [**TorchAudio**](https://pytorch.org/audio/stable/index.html) — PyTorch’s domain library for everything audio. \n",
    "- **[TorchRec](https://pytorch.org/torchrec/)** — PyTorch’s newest in-built domain library for powering recommendation engines with deep learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "966a5d29-d2f4-4089-b3a7-6b400f695d20",
   "metadata": {},
   "source": [
    "### Computer Vision \n",
    "\n",
    "> **Note:** For an in-depth overview of computer vision in PyTorch, see [03. PyTorch Computer Vision](https://www.learnpytorch.io/03_pytorch_computer_vision/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5353bfa2-d950-4f23-98c3-d29ff7c85e85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Base computer vision library\n",
    "import torchvision\n",
    "\n",
    "# Other components of TorchVision (premade datasets, pretrained models and image transforms)\n",
    "from torchvision import datasets, models, transforms "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d044f35e-068f-4664-a16c-b3150a842ce2",
   "metadata": {},
   "source": [
    "### Text and Natural Language Processing (NLP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "328b5026-fb0e-42f6-a50f-e4171a379c86",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Base text and natural language processing library\n",
    "import torchtext\n",
    "\n",
    "# Other components of TorchText (premade datasets, pretrained models and text transforms)\n",
    "from torchtext import datasets, models, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2409d640-a768-4423-a37c-3f4f2472b319",
   "metadata": {},
   "source": [
    "### Audio and Speech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "956edd5a-291b-41d5-b7fa-507a171171dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Base audio and speech processing library\n",
    "import torchaudio\n",
    "\n",
    "# Other components of TorchAudio (premade datasets, pretrained models and text transforms)\n",
    "from torchaudio import datasets, models, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cf19cae-f18b-4bb9-af99-56725a739df7",
   "metadata": {},
   "source": [
    "### Recommendation systems\n",
    "\n",
    "> **Note:** This library is currently in beta release, see the [GitHub page for installation](https://github.com/pytorch/torchrec#installation). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ddb24f63-13c9-406c-9608-f66042a99f87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Base recommendation system library \n",
    "# import torchrec\n",
    "\n",
    "# # Other components of TorchRec\n",
    "# from torchrec import datasets, models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0dc7202-271a-4473-94fc-4945319bda37",
   "metadata": {},
   "source": [
    "## Device-agnostic code (using PyTorch on CPU, GPU or MPS)\n",
    "\n",
    "Much of deep learning involves computing on tensors. \n",
    "\n",
    "Computing on tensors generally happens much faster on GPUs (graphics processing units, typically from NVIDIA) than CPUs (computer processing units).\n",
    "\n",
    "MPS stands for \"Metal Performance Shader\" which is Apple's GPU (M1, M1 Pro, M2 etc).\n",
    "\n",
    "It is advised to perform training on the fastest piece of hardware you have available, which will generally be: NVIDIA GPU (`\"cuda\"`) > MPS device (`\"mps\"`) > CPU (`\"cpu\"`).\n",
    "\n",
    "* For more on seeing how to get [PyTorch to run on NVIDIA GPU (with CUDA)](https://pytorch.org/docs/stable/cuda.html), see [00. PyTorch Fundamentals section 2: getting PyTorch to run on the GPU](https://www.learnpytorch.io/00_pytorch_fundamentals/#2-getting-pytorch-to-run-on-the-gpu).\n",
    "* For more on running PyTorch using an MPS backend (running PyTorch on Mac GPUs) [see the PyTorch documentation](https://pytorch.org/docs/stable/notes/mps.html). \n",
    "\n",
    "> **Note:** It is advised to setup device-agnostic code at the start of your workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "081f561d-63f7-4609-86eb-868750eb8b8a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using device: mps\n"
     ]
    }
   ],
   "source": [
    "# Setup device-agnostic code \n",
    "if torch.cuda.is_available():\n",
    "    device = \"cuda\" # NVIDIA GPU\n",
    "elif torch.backends.mps.is_available():\n",
    "    device = \"mps\" # Apple GPU\n",
    "else:\n",
    "    device = \"cpu\" # Defaults to CPU if NVIDIA GPU/Apple GPU aren't available\n",
    "\n",
    "print(f\"Using device: {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "208f1f37-470f-486a-9a71-b7128ae295a9",
   "metadata": {},
   "source": [
    "### Sending a tensor to target device\n",
    "\n",
    "You can move objects (models and tensors) in PyTorch to different devices via the `.to(\"device_name\")` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7dc6c29a-5d0a-45e1-9dde-547356e1366e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu\n",
      "mps:0\n"
     ]
    }
   ],
   "source": [
    "# Create a tensor \n",
    "x = torch.tensor([1, 2, 3]) \n",
    "print(x.device) # defaults to CPU \n",
    "\n",
    "# Send tensor to target device\n",
    "x = x.to(device)\n",
    "print(x.device) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d27d062-281b-4628-9374-c276640aceb8",
   "metadata": {},
   "source": [
    "## Setting random seeds\n",
    "\n",
    "A lot of machine learning and deep learning involves taking random numbers in tensors and then shaping those random numbers to find/represent patterns in real data. \n",
    "\n",
    "However, sometimes you'll want \"reproducible\" randomness.\n",
    "\n",
    "To do so, you can set the random seeds, see [Reproducibility (trying to take the random out of random)](https://www.learnpytorch.io/00_pytorch_fundamentals/#reproducibility-trying-to-take-the-random-out-of-random) for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "de557ce2-60df-4945-a57a-a8dc5e7de756",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor A:\n",
      "tensor([[0.8823, 0.9150, 0.3829, 0.9593],\n",
      "        [0.3904, 0.6009, 0.2566, 0.7936],\n",
      "        [0.9408, 0.1332, 0.9346, 0.5936]])\n",
      "\n",
      "Tensor B:\n",
      "tensor([[0.8823, 0.9150, 0.3829, 0.9593],\n",
      "        [0.3904, 0.6009, 0.2566, 0.7936],\n",
      "        [0.9408, 0.1332, 0.9346, 0.5936]])\n",
      "\n",
      "Does Tensor A equal Tensor B? (anywhere)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[True, True, True, True],\n",
       "        [True, True, True, True],\n",
       "        [True, True, True, True]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# Set the random seed (you can set this to any number you like, it will \"flavour\"\n",
    "# the randomness with that number.\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# Create two random tensors\n",
    "random_tensor_A = torch.rand(3, 4)\n",
    "\n",
    "torch.manual_seed(42) # set the seed again (try commenting this out and see what happens)\n",
    "random_tensor_B = torch.rand(3, 4)\n",
    "\n",
    "print(f\"Tensor A:\\n{random_tensor_A}\\n\")\n",
    "print(f\"Tensor B:\\n{random_tensor_B}\\n\")\n",
    "print(f\"Does Tensor A equal Tensor B? (anywhere)\")\n",
    "random_tensor_A == random_tensor_B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3ea594e-6ae6-4637-97bb-e52cd537df64",
   "metadata": {},
   "source": [
    "You can also set the random seed on the GPU (CUDA devices)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b2499098-9311-40b2-8ff3-63d433064aaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set random seed on GPU\n",
    "torch.cuda.manual_seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3efd5fba-9578-481b-850e-a968ade7700a",
   "metadata": {},
   "source": [
    "## Neural Networks\n",
    "\n",
    "PyTorch has a very comprehensive library of pre-built neural network components (many of these are referred to as \"modules\" in the PyTorch ecosystem).\n",
    "\n",
    "At a fundamental level neural networks are stacks of *layers*. Each of these layers performs some kind of operation on an input and produces an output.\n",
    "\n",
    "How these layers stack together will depend on the problem you're working on.\n",
    "\n",
    "One of the most active areas of research in machine learning is how to stack neural network layers together (and the best answer to this is constantly changing). \n",
    "\n",
    "The vast majority of neural network components in PyTorch are contained within the [`torch.nn` package](https://pytorch.org/docs/stable/nn.html) (`nn` is short for neural networks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "888b039b-c4ad-4f47-bad7-75ac8ca7db12",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "497528d9-9fef-4323-8438-b7a298ed7ea3",
   "metadata": {},
   "source": [
    "### Linear layers\n",
    "\n",
    "PyTorch has several in-built [linear layers](https://pytorch.org/docs/stable/nn.html#linear-layers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "79047b27-3752-4e36-b7a9-a8e95bac20c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a linear layer with 10 in features and out features\n",
    "linear_layer = nn.Linear(in_features=10,\n",
    "                         out_features=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4972c605-39b4-43e4-aba4-02d9527cbbde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an Identity layer\n",
    "identity_layer = nn.Identity()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9115ee4f-67a1-4adb-92e1-7604ccbb4b62",
   "metadata": {},
   "source": [
    "### Convolutional Layers (for making Convolutional Neural Networks or CNN's)\n",
    "\n",
    "PyTorch has [several in-built convolutional layers](https://pytorch.org/docs/stable/nn.html#convolution-layers).\n",
    "\n",
    "Naming of convolutional layers usually follows `torch.nn.ConvXd` where `X` can be a value of `1`, `2` or `3`.\n",
    "\n",
    "The `X` value represents the number of dimensions the convolution will operate over, for example, `1` for singular dimension text, `2` for two dimension images (height x width) and `3` for 3D objects such as video (video is considered a series of images with a time dimension, height x width x time).\n",
    "\n",
    "> **Note:** You can see more on building convolutional neural networks for computer vision with PyTorch in [03. PyTorch Computer Vision section 7.2: building a convolutional neural network (CNN)](https://www.learnpytorch.io/03_pytorch_computer_vision/#7-model-2-building-a-convolutional-neural-network-cnn)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "882a8c21-cc87-4ebe-9a03-68f9765647c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Conv1d layer (often used for text with a singular dimension)\n",
    "conv1d = nn.Conv1d(in_channels=1,\n",
    "                   out_channels=10,\n",
    "                   kernel_size=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "93d96033-75e4-4695-ad39-c488bb0a5038",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Conv2d layer (often used for images with Height x Width dimensions)\n",
    "conv2d = nn.Conv2d(in_channels=3, # 3 channels for color images (red, green, blue)\n",
    "                   out_channels=10,\n",
    "                   kernel_size=3)                   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b0b697f0-b80b-4dc7-a7d9-b771dad6c171",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Conv3d layer (often used for video with Height x Width x Time dimensions)\n",
    "conv3d = nn.Conv3d(in_channels=3,\n",
    "                   out_channels=10,\n",
    "                   kernel_size=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6042dabb-9f18-4cd1-916b-b0d864048b7e",
   "metadata": {},
   "source": [
    "### Transformer Layers (for making Transformer models)\n",
    "\n",
    "PyTorch has in-built Transformer layers as described in the paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762).\n",
    "\n",
    "Using in-built PyTorch Transformer layers has the benefit of potential speedups thanks to [PyTorch's BetterTransformer](https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/).\n",
    "\n",
    "> **Note:** You can see the use of PyTorch's in-built Transformer layers to build a Vision Transformer in [08. PyTorch Paper Replicating](https://www.learnpytorch.io/08_pytorch_paper_replicating/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0bcab8b8-defc-4d8c-8d75-c7826f200dd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Transformer model (model based on the paper \"Attention Is All You Need\" - https://arxiv.org/abs/1706.03762)\n",
    "transformer_model = nn.Transformer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ba384ef7-d6f4-4969-80da-f477634ea96f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a single Transformer encoder cell\n",
    "transformer_encoder = nn.TransformerEncoderLayer(d_model=768, # embedding dimension\n",
    "                                                 nhead=12) # number of attention heads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6e761141-0967-4804-a71b-03994132f865",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stack together Transformer encoder cells\n",
    "transformer_encoder_stack = nn.TransformerEncoder(encoder_layer=transformer_encoder, # from above\n",
    "                                                  num_layers=6) # 6 Transformer encoders stacked on top of each other"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "afdc3a0f-b4c4-47a2-ab29-999e66b5ed08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a single Transformer decoder cell\n",
    "transformer_decoder = nn.TransformerDecoderLayer(d_model=768,\n",
    "                                                 nhead=12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0bf68ba3-2f4b-454f-a5fe-3707bf52fa9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stack together Transformer decoder cells\n",
    "transformer_decoder_stack = nn.TransformerDecoder(decoder_layer=transformer_decoder, # from above\n",
    "                                                  num_layers=6) # 6 Transformer decoders stacked on top of each other"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6397f85-2263-47b1-b720-c3d46bf0eaef",
   "metadata": {},
   "source": [
    "### Recurrent Layers (for making Recurrent Neural Networks or RNN's)\n",
    "\n",
    "PyTorch has in-built support for [Recurrent Neural Network layers](https://pytorch.org/docs/stable/nn.html#recurrent-layers) such as [long short-term memory (LSTM)](https://en.wikipedia.org/wiki/Long_short-term_memory) and [gated recurrent unit (GRU)](https://en.wikipedia.org/wiki/Gated_recurrent_unit).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "aabbfdc0-3673-42c8-b075-62664cc4be85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a single LSTM cell\n",
    "lstm_cell = nn.LSTMCell(input_size=10, # can adjust as necessary\n",
    "                        hidden_size=10) # can adjust as necessary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "df67845c-55b5-4e67-9058-699075644b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stack together LSTM cells\n",
    "lstm_stack = nn.LSTM(input_size=10,\n",
    "                     hidden_size=10,\n",
    "                     num_layers=3) # 3 single LSTM cells stacked on top of each other"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e421579e-ab99-4b5f-be73-2a7f11f70d36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a single GRU cell\n",
    "gru_cell = nn.GRUCell(input_size=10, # can adjust as necessary\n",
    "                      hidden_size=10) # can adjust as necessary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "083284c1-501a-466a-b4e9-901a079f958c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stack together GRU cells\n",
    "gru_stack = nn.GRU(input_size=10, \n",
    "                   hidden_size=10,\n",
    "                   num_layers=3) # 3 single GRU cells stacked on top of each other "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5087611c-aae7-41fe-8397-03e01c885ade",
   "metadata": {},
   "source": [
    "### Activation Functions \n",
    "\n",
    "[Activation functions](https://en.wikipedia.org/wiki/Activation_function) often go between layers in a neural network to add non-linear (non-straight) capabilities to linear (straight) functions.\n",
    "\n",
    "In essence, a neural network is often comprised of a large amount of linear and non-linear functions.\n",
    "\n",
    "PyTorch has [several non-linear activation functions](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity) built into `torch.nn`.\n",
    "\n",
    "Some of the most common are:\n",
    "* `nn.ReLU` - also known as [rectified linear unit](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)).\n",
    "* `nn.Sigmoid` - also known as the [sigmoid function](https://en.wikipedia.org/wiki/Sigmoid_function).\n",
    "* `nn.Softmax` - also known as the [softmax function](https://en.wikipedia.org/wiki/Softmax_function).\n",
    "\n",
    "> **Note:** See [02. PyTorch Neural Network Classification section 6: non-linearity, the missing piece](https://www.learnpytorch.io/02_pytorch_classification/#6-the-missing-piece-non-linearity) for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "38581ec6-ed57-4c2f-89bb-f99193939cac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ReLU\n",
    "relu = nn.ReLU()\n",
    "\n",
    "# Sigmoid\n",
    "sigmoid = nn.Sigmoid()\n",
    "\n",
    "# Softmax\n",
    "softmax = nn.Softmax()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf38e1c3-1159-439d-ad3b-0db1b7fa3d12",
   "metadata": {},
   "source": [
    "### Loss Functions\n",
    "\n",
    "A loss function measures how *wrong* your model is. As in, how far are its predictions off where they should be.\n",
    "\n",
    "Ideally, with training, data and an optimization function, this loss value goes as low as possible.\n",
    "\n",
    "Loss functions in PyTorch (and deep learning in general) are also often referred to as: criterion, cost function.\n",
    "\n",
    "PyTorch has [several loss functions](https://pytorch.org/docs/stable/nn.html#loss-functions) built into `torch.nn`.\n",
    "\n",
    "And some of the most common are:\n",
    "* [`nn.L1Loss`](https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html#torch.nn.L1Loss) - also referred to as MAE or [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error) (this loss is often used for regression problems or predicting a number such as the price of houses).\n",
    "* [`nn.MSELoss`](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss) - also referred to as L2Loss or [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error) (this loss is often used for regression problems or predicting a number such as the price of houses).\n",
    "* [`nn.BCEWithLogitsLoss`](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss) - also known as [binary cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) this loss function is often used for binary classification problems (classifying something as one thing or another).\n",
    "* [`nn.CrossEntropyLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss) - this loss function is often used for multi-class classification problems (classifying something as one thing or another)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7ffe3146-759e-4f2f-aeea-ec1ecb506ad2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# L1Loss\n",
    "loss_fn = nn.L1Loss() # also known as MAE or mean absolute error\n",
    "\n",
    "# MSELoss\n",
    "loss_fn = nn.MSELoss() # also known as MSE or mean squared error\n",
    "\n",
    "# Binary cross entropy (for binary classification problems)\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "\n",
    "# Cross entropy (for multi-class classification problems)\n",
    "loss_fn = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5746cf2-2ff9-4621-a681-df36cac3beb1",
   "metadata": {},
   "source": [
    "### Optimizers \n",
    "\n",
    "An optimizer's job is to change the neural network weights in such a way that it reduces the loss function value.\n",
    "\n",
    "PyTorch has [several optimization functions](https://pytorch.org/docs/stable/optim.html) built into the `torch.optim` module. \n",
    "\n",
    "Two of the main optimizer functions include:\n",
    "* [`torch.optim.SGD(lr=0.1, params=model.parameters())`](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD) - SGD also known as [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) (`lr` stands for \"learning rate\", the multiplier of how much to modify neural network weights at each step, small value = small adjustments, big value = big adjustments).\n",
    "* [`torch.optim.Adam(lr=0.001, params=model.parameters())`](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam) - The Adam optimizer (`params` stands for \"model parameters\", in other words, the model parameters/weights you'd like the optimization function to optimize during training)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e6ec2382-d11d-40d4-afb7-fec01f4abccb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a baseline model\n",
    "model = nn.Transformer()\n",
    "\n",
    "# SGD (stochastic gradient descent)\n",
    "optimizer = torch.optim.SGD(lr=0.1, # set the learning rate (required)\n",
    "                            params=model.parameters()) # tell the optimizer what parameters to optimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "75a27e05-12fe-42f1-83aa-1e488f594373",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a baseline model\n",
    "model = nn.Transformer()\n",
    "\n",
    "# Adam optimizer\n",
    "optimizer = torch.optim.Adam(lr=0.001, # set the learning rate (required)\n",
    "                             params=model.parameters()) # tell the optimizer what parameters to optimize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e4bef22-4a74-487e-b8c6-18cb90bb0964",
   "metadata": {},
   "source": [
    "## End-to-end example workflow\n",
    "\n",
    "Let's put everything together in a quick end-to-end workflow.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/01_a_pytorch_workflow.png\" width=950 alt=\"a PyTorch workflow from data to building a model to fitting a model to evaluating the model\"/> \n",
    "\n",
    "This workflow has been taken from [01. PyTorch Workflow Fundamentals](https://www.learnpytorch.io/01_pytorch_workflow/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24805c73-7558-4d27-aba1-a78fce5312ce",
   "metadata": {},
   "source": [
    "### Create data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "7fed19d7-0e4e-42e8-ad80-8504033b3f6d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0.0000],\n",
       "         [0.0200],\n",
       "         [0.0400],\n",
       "         [0.0600],\n",
       "         [0.0800],\n",
       "         [0.1000],\n",
       "         [0.1200],\n",
       "         [0.1400],\n",
       "         [0.1600],\n",
       "         [0.1800]]),\n",
       " tensor([[0.3000],\n",
       "         [0.3140],\n",
       "         [0.3280],\n",
       "         [0.3420],\n",
       "         [0.3560],\n",
       "         [0.3700],\n",
       "         [0.3840],\n",
       "         [0.3980],\n",
       "         [0.4120],\n",
       "         [0.4260]]))"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create *known* parameters\n",
    "weight = 0.7\n",
    "bias = 0.3\n",
    "\n",
    "# Create data\n",
    "start = 0\n",
    "end = 1\n",
    "step = 0.02\n",
    "X = torch.arange(start, end, step).unsqueeze(dim=1) # data\n",
    "y = weight * X + bias # labels (want model to learn from data to predict these)\n",
    "\n",
    "X[:10], y[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "7dbfae76-edbf-470b-8686-8e12e30bbc8b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40, 40, 10, 10)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create train/test split\n",
    "train_split = int(0.8 * len(X)) # 80% of data used for training set, 20% for testing \n",
    "X_train, y_train = X[:train_split], y[:train_split]\n",
    "X_test, y_test = X[train_split:], y[train_split:]\n",
    "\n",
    "len(X_train), len(y_train), len(X_test), len(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b525ca1-bcc3-48c9-8b6b-ead556a50e9c",
   "metadata": {},
   "source": [
    "### Create a model\n",
    "\n",
    "Two main ways to create a model in PyTorch: \n",
    "1. Subclass `torch.nn.Module` - more code but can be very flexible, models that subclass `torch.nn.Module` must implement a `forward()` method.\n",
    "2. Use `torch.nn.Sequential` - less code but less flexibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "0f102f8c-ad12-48b4-bfc6-bde5eb945bc8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(LinearRegressionModel(\n",
       "   (linear_layer): Linear(in_features=1, out_features=1, bias=True)\n",
       " ),\n",
       " OrderedDict([('linear_layer.weight', tensor([[0.5025]])),\n",
       "              ('linear_layer.bias', tensor([-0.0722]))]))"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from torch import nn\n",
    "\n",
    "# Option 1 - subclass torch.nn.Module\n",
    "class LinearRegressionModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # Use nn.Linear() for creating the model parameters\n",
    "        self.linear_layer = nn.Linear(in_features=1, \n",
    "                                      out_features=1)\n",
    "    \n",
    "    # Define the forward computation (input data x flows through nn.Linear())\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        return self.linear_layer(x)\n",
    "\n",
    "model_0 = LinearRegressionModel()\n",
    "model_0, model_0.state_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb37ecd4-6258-404d-8dc3-149cd8540af3",
   "metadata": {},
   "source": [
    "Now let's create the same model as above but using `torch.nn.Sequential`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e2c357fd-68ef-428e-90ae-ed225dad2e4d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Sequential(\n",
       "   (0): Linear(in_features=1, out_features=1, bias=True)\n",
       " ),\n",
       " OrderedDict([('0.weight', tensor([[0.9905]])), ('0.bias', tensor([0.9053]))]))"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from torch import nn\n",
    "\n",
    "# Option 2 - use torch.nn.Sequential\n",
    "model_1 = torch.nn.Sequential(\n",
    "    nn.Linear(in_features=1,\n",
    "              out_features=1))\n",
    "\n",
    "model_1, model_1.state_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11692983-6190-4d51-9d4e-5293f4205988",
   "metadata": {},
   "source": [
    "### Setup loss function and optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "5c3d1c6d-a60f-4642-994d-7e0cb822164b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create loss function\n",
    "loss_fn = nn.L1Loss()\n",
    "\n",
    "# Create optimizer\n",
    "optimizer = torch.optim.SGD(params=model_1.parameters(), # optimize newly created model's parameters\n",
    "                            lr=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0b2f55e-8037-4942-8b74-0a09a3f4b007",
   "metadata": {},
   "source": [
    "### Create a training/testing loop\n",
    "\n",
    "Our goal is to reduce the loss of our model (how much our model's predictions are different to the actual data).\n",
    "\n",
    "If our training/testing loops are implemented right and the model is capable of learning patterns in the data, the training and test losses should go down.\n",
    "\n",
    "See the following for steps in a PyTorch training loop:\n",
    "* [PyTorch optimization loop song](https://youtu.be/Nutpusq_AFw)\n",
    "* [PyTorch Workflow Fundamentals Section 3: Training Loop](https://www.learnpytorch.io/01_pytorch_workflow/#pytorch-training-loop)\n",
    "* [PyTorch Workflow Fundamentals Section 3: Testing Loop](https://www.learnpytorch.io/01_pytorch_workflow/#pytorch-testing-loop)\n",
    "* [PyTorch Workflow Fundamentals Section 6.3: Training](https://www.learnpytorch.io/01_pytorch_workflow/#63-training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "0064882b-5fc7-4479-ad81-0dc0686ea711",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 100 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 200 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 300 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 400 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 500 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 600 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 700 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 800 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
      "Epoch: 900 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# Set the number of epochs \n",
    "epochs = 1000 \n",
    "\n",
    "# Put data on the available device\n",
    "# Without this, an error will happen (not all data on target device)\n",
    "X_train = X_train.to(device)\n",
    "X_test = X_test.to(device)\n",
    "y_train = y_train.to(device)\n",
    "y_test = y_test.to(device)\n",
    "\n",
    "# Put model on the available device\n",
    "# With this, an error will happen (the model is not on target device)\n",
    "model_1 = model_1.to(device)\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    ### Training\n",
    "    model_1.train() # train mode is on by default after construction\n",
    "\n",
    "    # 1. Forward pass\n",
    "    y_pred = model_1(X_train)\n",
    "\n",
    "    # 2. Calculate loss\n",
    "    loss = loss_fn(y_pred, y_train)\n",
    "\n",
    "    # 3. Zero grad optimizer\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    # 4. Loss backward\n",
    "    loss.backward()\n",
    "\n",
    "    # 5. Step the optimizer\n",
    "    optimizer.step()\n",
    "\n",
    "    ### Testing\n",
    "    model_1.eval() # put the model in evaluation mode for testing (inference)\n",
    "    # 1. Forward pass\n",
    "    with torch.inference_mode():\n",
    "        test_pred = model_1(X_test)\n",
    "    \n",
    "        # 2. Calculate the loss\n",
    "        test_loss = loss_fn(test_pred, y_test)\n",
    "\n",
    "    if epoch % 100 == 0:\n",
    "        print(f\"Epoch: {epoch} | Train loss: {loss} | Test loss: {test_loss}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af70942c-b2de-47ae-a266-d8097a124600",
   "metadata": {},
   "source": [
    "## Extras\n",
    "\n",
    "The above list is not exhaustive.\n",
    "\n",
    "Here are some good places to find out more:\n",
    "\n",
    "* [PyTorch official cheatsheet](https://pytorch.org/tutorials/beginner/ptcheat.html).\n",
    "* [Zero to Mastery Learn PyTorch course](https://dbourke.link/ZTMPyTorch) - a comprehensive yet beginner-friendly deep dive into using PyTorch for deep learning all the way from the fundamentals to deploying a model to the real-world so other people can use it.\n",
    "* [PyTorch performance tuning guide](https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html) - a resource from the PyTorch team on how to tune performance of PyTorch models.\n",
    "* [PyTorch Extra Resources](https://www.learnpytorch.io/pytorch_extra_resources/) - a curated list of helpful resources to extend PyTorch and learn more about the engineering side of things around deep learning.\n",
    "* [Effective PyTorch by vahidk](https://github.com/vahidk/EffectivePyTorch) - a GitHub repo with a fantastic overview of some of the main functionality in PyTorch in a straight-forward manner."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
